Papers with threat model
Effective Red-Teaming of Policy-Adherent Agents (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Model (LLM)-based agents are increasingly used in domains with strict policies, such as refund eligibility or cancellation rules. |
| Approach: | They propose a multi-agent red-teaming system that leverages policy-aware persuasive strategies to undermine a policy-adherence agent in a customer-service scenario. |
| Outcome: | The proposed model outperforms jailbreak methods and tau-break to assess agent's robustness against manipulative user behavior. |
PoisonedParrot: Subtle Data Poisoning Attacks to Elicit Copyright-Infringing Content from Large Language Models (2025.naacl-long)
Copied to clipboard
Michael-Andrei Panaitescu-Liess, Pankayaraj Pathmanathan, Yigitcan Kaya, Zora Che, Bang An, Sicheng Zhu, Aakriti Agrawal, Furong Huang
| Challenge: | PoisonedParrot is the first stealthy data poisoning attack that induces an LLM to generate copyrighted content even when the model has not been directly trained on the copyright material. |
| Approach: | They propose a stealthy data poisoning attack that induces an LLM to generate copyrighted content even when it has not been directly trained on the copyright material. |
| Outcome: | The proposed model induces an LLM to generate copyrighted content with no discernible side effects and is surprisingly effective at priming the model to generate content with little side effects. |
RedHerring Attack: Testing the Reliability of Attack Detection (2025.emnlp-main)
Copied to clipboard
| Challenge: | In response to adversarial text attacks, attack detection models have been proposed and shown to successfully identify text modified by adversaries. |
| Approach: | They propose to modify a text to cause the detection model to predict an attack while keeping the classifier correct. |
| Outcome: | The proposed model drops detection accuracy between 20 - 71 points while maintaining (or improving) classifier accuracy. |
TrojanStego: Your Language Model Can Secretly Be A Steganographic Privacy Leaking Agent (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing work has focused on the (un)intended leakage of sensitive information through LLM outputs. |
| Approach: | They propose a threat model that embeds context information into natural-looking outputs via linguistic steganography without requiring explicit control over inference inputs. |
| Outcome: | The proposed model transmits 32-bit secrets with 87% accuracy on held-out prompts and can reach over 97% accuracy using majority voting across three generations. |